Analysis  of  Decision  Making  Skills  for  Large  Scale 

Disaster  Response 


Charles  Rose,  Robert  Seater,  Adam  Norige 
Humanitarian  Assistance  and  Disaster  Relief  Systems 
MIT  Lincoln  Laboratory 


Abstract — A  large  scale  disaster  such  as  the  detonation  of 
an  improvised  nuclear  device  (IND)  in  a  U.S.  city  would  pose 
significant  response  challenges  for  all  levels  of  government, 
private  organizations,  and  the  general  public.  Public  officials 
and  emergency  managers  would  face  difficult  and  high  impact 
choices  throughout  the  response  effort,  and  they  must  prepare 
to  make  timely  and  key  decisions  throughout  the  effort.  Decision 
making  preparation  may  involve  more  than  technical  training 
and  resources.  It  may  extend  to  emergency  managers  being 
cognitively  and  emotionally  prepared  for  the  situations  they  may 
face.  This  paper  presents  the  first  step  toward  the  larger  goal 
of  developing  alternative  disaster  preparedness  training  methods 
that  teach  effective  decision  making.  The  project  team  interviewed 
highly  experienced,  disaster  response  professionals  and  analyzed 
decisions  they  emphasized  as  being  both  important  and  difficult 
during  an  IND  response.  The  respondents  also  identified  the 
critical  skills  needed  to  make  those  decisions  effectively.  This 
paper  reports  on  the  findings  and  analysis  of  specific  decisions 
and  skills  required  for  an  IND  response. 

I.  Introduction 

The  overall  project  [1]  addresses  the  question  of  how 
government  agencies  can  ensure  that  emergency  managers 
have  the  appropriate  knowledge,  skills,  and  abilities  to  operate 
effectively  during  an  incident  of  unprecedented  size,  scope, 
and  complexity,  such  as  an  IND  detonation  [2]  [3].  To  help 
answer  this  question,  we  are  exploring  the  use  of  gaming  as 
a  skill  enhancement  and  evaluation  technique  to  target  ’soft 
skills,’  which  include  personality  traits,  emotional  intelligence, 
critical  thinking,  and  inter-personal  skills,  as  well  as  improving 
the  retention  of  training  ’hard  skills,’  such  as  technical  knowl¬ 
edge,  familiarity  with  best  practice,  and  knowledge  about  an 
organization’s  structure  and  operation.  The  project  evaluates 
the  viability  of  expanding  the  use  of  serious  games  to  augment 
classroom  training,  tabletop  and  full  scale  exercise,  and  actual 
disaster  experience  to  better  cover  the  critical  skills  needed 
for  effective  emergency  management.  This  report  documents 
the  first  phase  of  the  project:  analyzing  the  key  decisions  and 
skills  relevant  to  the  response  effort  for  an  IND  detonation. 
Subsequent  phases  of  the  project  will  build  on  this  catalog 
of  decisions  and  skills  to  evaluate,  design,  and  integrate  new 
training  and  evaluation  mechanisms  that  could  better  prepare 
emergency  managers  to  respond  to  catastrophic  incidents. 

Serious  games  are  used  throughout  the  Department  of 
Homeland  Security  (DHS)  and  other  government  agencies 
to  support  training,  evaluation,  analysis,  and  technology  ex¬ 
ploration.  Those  techniques  have  found  successful  niches, 
but  their  wider  applicability  faces  several  practical  barriers. 
Those  barriers  can  potentially  be  overcome  by  drawing  on 
techniques,  technologies,  and  expertise  from  the  entertainment 


game  industry.  The  recent  surge  of  activity  in  attempting  to  do 
so  has  had  successes  [4]  [5]  but  is  hampered  by  the  lack  of 
a  systematic  approach  for  reliably  building  games  to  match  a 
given  application. 

A.  Learning  Critical  Skills 

It  is  difficult  and  time  consuming  to  gain  expertise  within 
a  specific  job  or  profession  when  the  required  skills  are  chal¬ 
lenging  to  learn  and  opportunities  to  gain  actual  experience  are 
infrequent.  This  challenge  is  especially  prevalent  in  the  field 
of  radiological  and  nuclear  (rad/nuc)  emergency  management 
where  real  world  instances  are  extremely  rare  and  many  details 
are  theoretical.  Currently,  emergency  management  personnel 
hone  their  professional  competencies  through  a  combination  of 
classroom  and  on-the-job  learning.  Classroom-based  learning 
provides  professionals  with  the  fundamentals  of  their  posi¬ 
tion  requirements,  focusing  on  gaining  technical  knowledge, 
response  terminology,  and  incident  organization  fundamentals, 
including  the  National  Incident  Management  System  (NIMS) 
and  Incident  Command  System  (ICS).  On-the-job  learning  and 
experience-based  job  requirements  help  ensure  that  personnel 
have  learned  relevant  skills  by  successfully  tackling  real  prob¬ 
lems  and  challenges. 

The  combination  of  classroom  and  on-the-job  learning  can 
be  effective  and  can  establish  expertise  and  skill;  however 
there  are  shortcomings  with  each  method  individually,  and 
even  with  both  methods  combined.  Classroom-style  methods 
struggle  to  teach  and  help  students  retain  abstract  behavioral 
competencies.  On-the-job  learning  can  be  highly  effective, 
but  for  complex  professions  like  emergency  management  it 
can  be  a  slow  method  of  gaining  expertise,  especially  when 
incidents  are  rare  (as  is  the  case  for  large  scale  disasters).  Some 
incident  commanders  and  senior-level  emergency  managers 
spend  decades  becoming  proficient.  On-the-job  learning  is 
challenging  to  quantify  in  terms  of  improved  skill  and  per¬ 
formance,  and  it  is  hard  to  determine  if  experience  and  skills 
gained  in  one  type  of  situation  transfer  to  similar  situations 
(for  example,  between  disasters  of  different  types,  of  different 
scales,  or  in  different  locations).  To  be  better  prepared  for  a 
catastrophic  response,  new  and  innovative  methods  are  needed 
to  develop  emergency  managers’  decision  making  skills. 

The  nature  of  the  skills  to  be  learned  complicates  the  task  of 
cultivating  skilled  experts.  Within  emergency  management  as  a 
whole,  the  subjects  of  some  of  our  interviews  postulated  that 
the  distinguishing  competencies  of  successful  personnel  are 
more  behavioral  than  knowledge-based;  the  analysis  described 
in  this  report  supports  this  speculation.  Usually,  the  most 
adept  emergency  managers  and  incident  commanders  possess 
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a  refined  set  of  behavioral  competencies,  such  as  the  ability  to 
improvise,  make  correct  inferences,  and  to  selectively  extract 
useful  information  to  enhance  their  situational  awareness.  Such 
skills  can  be  built  up  through  lengthy  experience,  or  are 
perhaps  innate,  but  they  are  especially  hard  to  develop  and 
demonstrate  in  the  classroom  or  via  infrequent  field  experience. 
Waiting  for  candidate  emergency  managers  to  slowly  acquire 
and  demonstrate  the  skills  during  real  incidents  curbs  the 
available  talent  pool  and  reduces  confidence  that  key  decision 
makers  have  the  skills  to  handle  large  incident. 

Motivating  this  project  is  a  set  of  research  questions  arising 
from  the  discussion  above  that  skilled  emergency  managers  are 
successful  because  they  possess  certain  behavioral  competen¬ 
cies.  The  most  interesting  research  questions  are  listed  below: 
1)  Do  successful  emergency  managers,  especially  those  with  a 
lengthy  track  record  of  successful  field  experience,  actually 
value  and  exhibit  a  certain  set  of  behavioral  competencies 
and  are  these  skills  common  across  types  of  disasters?  2) 
Which  skills  are  the  most  important  to  each  level  or  type  of 
emergency  manager,  and  what  are  the  important  hard  decisions 
that  require  those  skills?  3)  Are  there  mechanisms  to  reliably 
determine  if  someone  already  possesses  those  critical  skills 
or  what  potential  they  exhibit  to  acquire  such  skills?  4)  Can 
the  learning  of  those  skills  be  accelerated  or  better  retained 
through  the  use  of  non-traditional  training  mechanisms,  such 
as  serious  games  or  game-like  exercises? 

The  interview  and  survey  analysis  described  in  this  re¬ 
port  target  research  questions  one  and  two  above,  laying  the 
groundwork  for  subsequent  phases  of  the  project  to  address 
new  evaluation  and  training  techniques  which  is  described  in 
the  following  subsection. 

B.  Serious  Games  in  the  U.S.  Government 

While  there  is  still  organization  stigma  attached  to  the  word 
’’gaming”  because  it  is  often  associated  with  entertainment, 
serious  games  are  already  widely  used  throughout  DHS,  the 
Department  of  Defense  (DoD),  and  other  agencies  to  improve 
employees’,  responders’,  and  warfighters’  skills  and  capabili¬ 
ties  to  execute  their  mission.  Serious  games  are  currently  used 
in  the  form  of  immersive  simulations,  live  exercises,  scenario 
based  training,  red  teaming,  and  war  gaming.  For  example: 

•  DHS  Transportation  Security  Administration  uses  a 
rehearsal  game  to  train  airport  checkpoint  screeners 
to  quickly  spot  dangerous  items,  to  identify  their 
best  employees,  and  to  provide  truth  data  for  training 
automated  detection  algorithms.  [6] 

•  Airline  pilots  and  air  traffic  controllers  gain  valuable 
flight  experience  from  detailed  simulators,  reducing 
the  cost  of  gaining  experience  by  augmenting  true 
flying  time  and  demonstrating  their  skills  in  a  low  risk 
environment. 

•  Emergency  responders  come  together  to  participate  in 
scenario-driven  exercises  to  practice  coordinating  and 
communicating,  and  to  build  organizational  ties  and 
social  networks  of  trust. 

•  Naval  officers  evaluate  the  relative  effectiveness  of 
different  command  structures  and  the  potential  uses 


of  emerging  technologies  through  war  games,  using  a 
mix  of  virtual  and  tabletop  materials.  [7] 

•  Cyber  security  analysts  often  use  a  competitive  ex¬ 
ercise  environment  with  a  red  team  attempting  to 
find  exploits  in  a  blue  team’s  defenses,  essentially 
crowdsourcing  human  ingenuity  for  finding  gaps. 

All  of  these  approaches  are  types  of  serious  games,  which 
can  be  defined  as  any  interactive  system  that  replicates  a 
key  dynamic  or  decision  space  from  the  real  world  in  a 
controlled,  artificial  environment  that  focuses  participants  on  a 
particular  aspect  of  the  domain.  Games  provide  success  criteria 
to  motivate  participants  to  learn,  exhibit,  or  explore  skills  and 
situations  selected  by  the  designer  and  of  interest  to  analysts. 
These  features  of  games  can  be  exploited  to  reduce  the  cost 
of  training,  supplement  real  world  experience,  and  target 
skills  and  situations  that  cannot  be  adequately  captured  in 
the  classroom.  While  existing  gaming  techniques  are  effective 
within  their  specific  applications,  there  is  a  potential  for  a 
much  broader  positive  impact.  That  broader  impact  is  currently 
limited  by  three  key  challenges  that  face  potential  users. 

CHALLENGE  1:  Historical  niches  and  focus  on  re¬ 
hearsal.  Most  current  serious  games  focus  on  rehearsal;  partic¬ 
ipants  walk  through  the  motions  of  known  procedures  and  best 
practices.  Rehearsal  is  a  very  important  part  of  preparedness, 
but,  as  our  analysis  indicates,  there  are  other  important  skills 
to  target  at  both  the  individual  and  organization  level.  For 
example,  dealing  with  unexpected  situations,  experimenting 
with  alternative  organizational  structures,  and  building  social 
networks  of  trust  are  sometimes  addressed  by  current  games, 
and  anecdotally  games  that  target  those  topics  are  very  helpful, 
but  generally  take  a  back  seat.  Naval  war  gaming  has  a  tradition 
of  emphasizing  such  dynamics  [7],  but  those  practices  are 
not  frequently  employed  by  DHS.  This  is  understandable, 
as  it  would  be  difficult  for  an  emergency  response  exercise 
designer  to  adapt  a  naval  war  game  to  their  situation  using 
current  tools  and  techniques.  A  more  systematic  and  data- 
driven  understanding  of  serious  games  could  offer  a  means  for 
identifying  when  a  game  is  an  appropriate  tool  to  use,  which 
type  of  game  would  best  suit  the  application,  and  provide 
guidance  for  adapting  previous  games  to  new  applications. 

CHALLENGE  2:  Lack  of  reusable  templates  and  in¬ 
frastructure.  The  design,  construction,  and  analysis  of  serious 
games  are  generally  ad  hoc  and  do  not  draw  on  a  common 
set  of  tools  or  techniques.  New  games  are  generally  created 
bottom-up,  both  in  terms  of  their  design  and  their  implemen¬ 
tation,  even  when  they  are  quite  similar  to  previous  games. 
A  catalog  of  game  templates  coupled  with  matching  reusable 
infrastructure  would  reduce  the  cost  of  creating  new  games  or 
adapting  old  games  to  new  situations. 

CHALLENGE  3:  High  operation  and  attendance  costs. 

Serious  games  currently  in  use  are  heavy  weight;  they  gener¬ 
ally  require  specialized  equipment,  a  considerable  time  com¬ 
mitment  by  participants  and  designers,  and  lengthy  manual 
analysis  of  the  outcomes.  Simulations  generally  require  partici¬ 
pants  to  visit  a  particular  location  to  use  specialized  equipment, 
limiting  the  volume  of  data  that  can  be  collected  (to  determine 
aggregate  patterns  and  trends).  Live  exercises  often  require 
participants  to  travel  to  a  common  location  and  remain  there 
for  several  days  in  a  row,  increasing  the  cost  of  such  training 


and  reducing  the  chance  that  an  agency  will  simultaneously 
send  all  its  best  personnel  for  training.  There  is  an  oppor¬ 
tunity  for  additional  styles  and  alternate  implementations  of 
serious  games  to  allow  asynchronous  or  remote  participation, 
to  require  less  specialized  equipment  to  operate,  to  better  target 
specific  skills  or  situations  of  interest,  and  to  overall  place  less 
burden  on  the  participants  and  designers. 

C.  Project  Mission 

This  project  seeks  to  analyze  potential  carry-overs  from 
the  commercial  gaming  industry  and  determine  their  applica¬ 
bility  to  addressing  existing  needs  and  gaps  within  DHS.  To 
accomplish  this  task,  the  project  aims  to: 

•  Identify  an  expanded  set  of  challenges  within  DHS 
that  could  be  addressed  with  games. 

•  Begin  to  build  a  systematic  classification  of  game 
styles  and  quantifiably  match  those  styles  to  suitable 
applications. 

•  Understand  the  types  of  collaborations  between  DHS 
and  private  industry  that  could  create  games  to  directly 
support  DHS  needs. 

The  project’s  initial  focus  is  on  an  IND  detonation  as 
a  concrete  example  of  a  DHS  mission  area  where  serious 
gaming  is  likely  to  be  productively  employed.  An  IND  det¬ 
onation  would  stress  DHS  and  emergency  response  systems 
and  resources,  so  an  analysis  of  the  relevant  decisions  and 
skills  of  such  an  incident  is  likely  to  identify  areas  of  need 
not  currently  being  sufficiently  met  by  existing  training  and 
preparation  methods.  Furthermore,  IND  incidents  are  low 
probability,  high  consequence  incidents,  making  preparation 
important  but  difficult  to  justify  when  it  is  costly.  Thus,  INDs 
represent  an  opportunity  for  lighter  weight  games  to  play  a 
valuable  role  in  preparation.  Lastly,  as  our  analysis  indicates 
that  an  IND  response  effort  requires  largely  the  same  skills 
as  other  emergency  response  but  with  unique  challenges.  The 
commonality  to  other  incidents  means  that  the  outcomes  of 
this  project  are  likely  to  apply  more  widely,  while  the  unique 
elements  will  drive  the  work  to  generate  more  general  game 
templates  (not  just  one-off  games)  that  are  flexible  and  that 
can  readily  be  adapted  to  other  scenarios. 

II.  Methodology 

The  first  step  in  designing  training  and  evaluation  mech¬ 
anisms  for  any  skill  set  is  to  identify  the  important  decisions 
and  corresponding  skills.  Since  no  IND  incidents  have  oc¬ 
curred  in  the  US,  there  is  no  direct  way  to  identify  those 
decisions  and  skills.  As  a  proxy  for  direct  experience  with 
an  IND,  we  turned  to  hazard  and  planning  documentation  and 
subject  matter  experts  (SMEs).  Existing  hazard  and  planning 
documentation  covers  the  nature  of  radiation  and  response 
plans;  however  our  primary  source  of  information  for  the 
key  decisions  and  skills  required  was  SMEs  with  experience 
creating  IND  incident  response  plans  and  participating  in  non- 
IND  large  scale  incidents. 

This  analysis  is  not  designed  to  be  a  comprehensive  guide 
to  an  IND  response.  The  focus  of  this  phase  of  the  project  is 
to  identify  trends  in  the  decisions  and  skills  SMEs  emphasized 


and  what  concerns  they  expressed  about  current  preparedness. 
These  results  and  our  interpretations  are  presented  in  the  fol¬ 
lowing  sections,  laying  the  groundwork  for  subsequent  phases 
of  the  project. 

Our  work  started  with  reviewing  relevant  IND  incident 
response  planning  documents,  including: 

•  Planning  Guidance  for  Response  to  a  Nuclear  Deto¬ 
nation  [2] 

•  Federal  Emergency  Management  Agency’s  (FEMA) 
IND  Response  Planning  Tool  [8] 

•  Key  Planning  Factors  for  Response  to  a  Nuclear 
Detonation  in  Houston,  Texas  [9] 

•  National  Response  Framework  Radiological/Nuclear 
Incident  Annex  [10] 

•  DHS  Strategy  for  Improving  the  National  Response 
and  Recovery  from  an  IND  Attack  [11] 

Next  we  interviewed  26  SMEs  in  the  fields  of  large  scale 
incidents  and  IND  incident  response  planning.  Our  interviews 
were  mainly  with  FEMA  Incident  Management  Assistance 
Team  (IMAT)  emergency  managers,  state  level  emergency 
managers,  federal  level  IND  planning  specialists,  state  level 
IND  planning  specialists,  and  radiological  health  experts.  The 
SMEs  have  extensive  experience  in  large  scale  emergency 
response  decision  making  at  the  local,  state,  and  national 
levels.  The  list  includes  high  level  coordinating  officers  from 
many  notable  major  disasters  such  as:  9/11  (2001),  Hurricane 
Katrina  (2005),  Haiti  Earthquake  (2010),  and  Hurricane  Sandy 
(2012).  While  there  is  a  wide  range  of  people  that  would  be 
involved  in  an  IND  detonation  incident  response  beyond  those 
that  were  targeted  for  this  study,  we  believe  that  the  group  of 
SMEs  chosen  for  this  study  is  representative  of  those  making 
the  most  critical  decisions. 

We  concluded  the  interview  process  when  we  no  longer 
heard  new  topics  from  the  interviewees,  giving  us  reasonable 
confidence  that  the  quantity  of  interviews  conducted  was 
sufficient  to  represent  the  dominant  views  of  this  type  of 
emergency  manager.  As  described  later  in  this  report,  there 
were  strong  consistencies  in  the  topics  that  the  SMEs  choose  to 
emphasize  when  asked  open  ended  questions.  The  interviewees 
responded  to  questions  by  phone  and  the  interview  sessions 
lasted  one  hour.  The  majority  of  the  interviews  had  a  single 
SME  present;  two  interviews  had  two  SMEs  each.  For  those 
two  interviews,  the  opinions  of  the  two  SMEs  were  combined 
into  a  single  entry  in  our  results.  A  project  summary  similar 
to  the  introduction  of  this  paper  was  provided  to  each  SME 
ahead  of  time.  The  majority  of  the  interview  time  included  a 
discussion  about  the  difficult  decisions  and  required  skills  in 
the  context  of  IND  and  large  scale  incidents.  The  discussion 
had  a  few  opened  ended  questions,  but  the  bulk  of  the  time 
was  open  for  the  interviewee  to  discuss  any  topics  relating 
to  rad/nuc  response  of  their  choosing.  Examples  of  the  open 
ended  questions  are: 

•  What  are  difficult  decisions  or  activities  in  an  IND  or 
large  scale  incident? 

•  What  are  difficult  positions  or  roles  in  an  IND  or  large 
scale  incident? 


What  skills  are  most  important  to  those  decisions? 


III.  Results 


•  How  comfortable  are  you  that  the  people  making  these 
decisions  have  the  skills  needed? 

•  Which  skills  are  hardest  to  teach,  select  for,  or  vali¬ 
date? 

This  interview  method  encouraged  discussions  that  al¬ 
lowed  each  SME  to  emphasize  the  most  challenging  and 
important  aspects  of  emergency  response  as  they  relate  to 
his/her  experiences  and  concerns.  The  topics  discussed  were 
spontaneously  emphasized  by  the  SMEs,  and  not  intended 
to  be  a  comprehensive  list.  The  interviews  were  meant  to 
verify,  prioritize,  and  augment  the  topics  discussed  in  more 
comprehensive  reports  (see  the  references  section)  not  to 
supersede  planning  documentation.  Our  final  step  was  to 
validate  the  findings  from  the  interviews.  This  was  done  by 
surveying  the  same  group  of  SMEs  plus  an  expanded  group  of 
similar  SMEs  to  rank  the  importance  and  level  of  concern  of 
information  collected  during  the  interview  process.  Answering 
questions  on  a  provided  list  of  topics  reduces  the  chance  that 
an  important  topic  was  accidentally  omitted  by  some  SMEs 
during  their  interview.  Note  that  a  ranking  approach  would 
not  have  been  appropriate  before  the  interviews,  as  ranking 
a  provided  list  would  not  allow  the  SMEs  to  express  their 
own  ideas  and  could  bias  them  toward  theories  we  already 
had.  However,  once  the  SMEs  provide  a  list  in  an  open  ended 
fashion,  a  more  structured  follow-on  analysis  can  be  useful 
and  complementary.  The  survey  specifically  asked  the  SMEs 
to  rank  the  importance  of  items  in  two  lists,  corresponding  to 
the  impactful  decisions  and  required  skills  of  an  IND  incident 
response.  When  the  SMEs  ranked  the  items,  they  received 
guidance  to  incorporate  their  opinions  about  two  topics:  the 
effectiveness  of  existing  decision  or  skill  training,  and  the 
importance  of  that  decision  or  skill. 


A.  Limitation  of  this  Analysis 

We  interviewed  26  experts  over  the  course  of  24  interviews, 
including  professionals  from  state,  local,  and  national  roles 
and  including  both  operational  decision  makers  and  domain 
experts.  Only  11  experts  responded  to  the  subsequent  survey, 
providing  us  only  a  limited  form  of  cross-validation  of  the  two 
approaches.  However,  there  was  strong  consistency  between 
the  survey  and  interviews,  increasing  our  confidence  in  the 
results  despite  the  sample  size.  Specific  discrepancies  are 
discussed  and  interpreted  in  the  full  manuscript  [12], 

Our  interview  approach  relies  on  several  key  assumptions. 
For  example,  we  interpret  emphasis  by  experienced  profession¬ 
als  as  being  reflective  of  the  actual  skills  that  are  important. 
We  assume  that  current  emergency  managers  have  the  requisite 
skills,  that  they  know  what  those  skills  are,  that  those  skills 
will  continue  to  be  relevant  in  the  future,  and  that  skills  for 
IND  incidents  carry  over  to  non-IND  incidents.  Our  intuition 
and  informal  feedback  from  the  interviewees  supports  these 
assumptions,  but  it  is  non-the-less  important  to  keep  these 
limitations  in  mind  when  interpreting  the  results  or  applying 
them  to  the  development  of  training  materials. 


A.  High  Impact  Decisions 

Figure  1  lists  all  nineteen  decisions  emphasized  by  three 
or  more  SMEs  as  being  critical  during  the  response  to  an  IND 
detonation  or  other  large  scale  disasters;  decisions  mentioned 
by  only  one  or  two  SMEs  were  omitted  as  non-representative. 
The  rest  of  this  section  provides  a  more  detailed  definition 
and  explanation  for  each  decision  based  on  the  information 
and  examples  directly  provided  by  the  SMEs.  The  level  of 
detail  devoted  to  each  item  in  this  section  reflects  the  detail 
provided  by  the  SMEs,  the  number  of  SMEs  who  discussed 
the  topic,  and  the  importance  placed  on  the  specific  decision 
by  the  SMEs  as  a  whole. 

Number  of  SMEs  that  Identified  the  Difficult  Decision 
(24  Total  Responses) 

How  to  allocate  finite  resources? 

How  to  phrase/disseminate  info  to  public  or  FRs? 

Whether  to  expose  first  responders  to  radiation? 

Where,  when,  and  how  to  evacuate? 

How  to  set  patient  triage  standards? 

Where  are  dangerous  fallout  and  hot  zones? 

Where  to  issue  shelter-in-place? 

How  to  develop  an  information  collection  plan? 

Where  to  stage  incoming  resources? 

Where  to  assemble  survivors? 

How  to  recognize  as  a  rad/nuc  incident? 

Where  are  the  damage  zones  (DZs)? 

Has  the  decision  making  staff  been  impacted? 

How  to  get  public  services  and  utilities  back  up? 

How  and  where  to  decontaminate  people? 

How  to  monitor  the  affected  population? 

How  to  maintain  public  confidence  and  trust? 

Do  FRs  restrict  who  can  leave/enter  areas? 

Single  detonation  or  is  there  a  secondary  device? 

Fig.  1.  Difficult  decisions  emphasized  by  SMEs 

The  majority  of  the  difficult  decisions  identified  by  the 
SMEs  are  required  in  some  form  for  other  large-scale  inci¬ 
dents;  however  the  SMEs  highlighted  many  specific  challenges 
and  complexities  of  a  rad/nuc  incident  in  making  nearly 
every  decision.  Their  emphasis  indicates  that  IND  incidents, 
and  more  generally  rad/nuc  emergencies,  do  present  distinct 
challenges. 

Table  I  shows  a  timeline  of  an  IND  incident  response 
planning  operations  from  the  IND  Response  Planning  Tool 
[8].  The  eight  most  common  difficult  decisions  as  identified 
by  the  SMEs  are  highlighted  in  red,  representing  decisions 
mentioned  in  at  least  one  third  of  the  interviews.  There  is  not  a 
direct  1:1  mapping  from  our  list  to  the  response  planning  list, 
but  all  of  the  eight  most  difficult  decisions  are  represented 
in  the  ten  items  highlighted  below  except  for  ’’developing 
an  information  collection  plan”  which  touches  many  of  the 
operations  in  the  0-6  hour  range.  Interestingly,  the  majority 


TABLE  I. 


IND  RESPONSE  PLANNING  TIMELINE  [8] 


0-1  Hour 

1-6  Hours 

6-24  Hours 

24-48  Hours 

48-72  Hours 

•  Assess  Comm. 
Availability 

•  Characterize  the 
Weapon 

•  Determine  Down 
Wind  Direction 

•  Establish  Area 
Command 

•  Give  Shelter-in- 
Place  Command 

•  Inform  the  Public 

•  Notify  Responders 

•  Recognize  Incident 
as  an  IND 

•  Assess  EOC 

Integrity 

•  Assess  Utilities 

•  Characterize  the 
Incident 

•  Determine  and 
Monitor  Weather 

•  Determine  Damage 
and  Fallout  Zones 

•  Determine  Medical 
Center  Conditions 

•  Establish  Dose 
Turnback  Levels 

•  Establish  Multiple 
Incident  Commands 

•  Perform  Operational 
Risk  Assessments 

•  Perform 

Radiological  Risk 
Assessments 

•  Plan  and  Allocate 
Resources 

•  Plan  and  Initialize 
Evacuation 

•  Preserve  Evidence 

•  Set  PPE  Standards 

•  Control  Access 

•  Control  Crowds 

•  Control  Fires 

•  Decontaminate 

People  and  Pets 

•  Initial  Mass 

Casualty  Triage 

•  Monitor  Population 

•  Perform  Sampling  & 
Laboratory  Analysis 

•  Search  Damaged  & 
Collapsed  Buildings 

•  Clear  Highways  / 
Freeways 

•  Establish  RTRS  and 
Other  Centers 

•  Integrate  Regional 
&  National  Assets 

•  Locate  &  Extract 
Survivors 

•  Distribute  Food  & 
Water  Supplies 

•  Establish  Medical 

Aid  Stations 

•  Manage  Fatalities  / 
Casualties 

•  Manage  Pets  & 
Service  Animals 

•  Recover  Decedents 

•  Relocate  Population 

•  Reunify  Families 

•  Disease  Prevention 
&  Control  Measures 

of  the  difficult  decisions  are  split  between  the  0-1  hour  and 
1-6  hour  ranges,  and  all  within  24  hours  after  detonation. 
This  provides  evidence  that  a  driving  factor  in  the  difficulty  of 
decisions  is  time  pressure. 

B.  Critical  Decision  Making  Skills 

Figure  2  lists  all  skills  emphasized  by  two  or  more  SMEs  as 
being  critical  during  the  response  to  an  IND  detonation  or  other 
large  scale  disaster.  Skills  mentioned  by  two  or  less  SMEs 
were  omitted  as  non-representative.  The  rest  of  this  section 
provides  a  more  detailed  definition  and  explanation  for  each 
skill  mentioned,  accompanied  by  examples  when  provided  by 
the  SMEs  for  clarification.  Similar  to  the  explanations  for  the 
key  decisions,  the  level  of  detail  of  the  skills  given  below  is 
reflective  of  the  detail  provided  by  the  SMEs  and  the  number  of 
SMEs  that  discussed  the  skills.  Our  interpretation  is  that  skills 
emphasized  by  more  SMEs  and  discussed  in  greater  detail  are 
likely  to  be  the  more  important  skills  or  skills  that  the  SMEs 
are  concerned  will  be  lacking  during  an  incident. 

Note  that  the  low  mention  rate  of  a  skill  does  not  mean 
that  the  skill  is  not  important.  A  low  rate  of  mention  might 
simply  indicate  one  of  the  following: 

•  The  SMEs  believe  that  everyone  under  consideration 
for  those  roles  has  that  skill  (i.e.  it  is  not  a  distinguish¬ 
ing  characteristic  of  the  best  emergency  management 
decision  makers). 


Number  of  SMEs  that  Identified  as  a  Skill  (24  Total  Responses) 


Make  a  timely  decision  with  limited  information 
Determine  and  use  reliable  information/sources 
Adapt  plans,  improvize  solutions 
Apply  resources  effectively  /  efficiently 
Synthesize  information,  gain  situational  awareness 


Forward  thinking,  able  to  adjust  decisions 
Focus  under  stress,  detach  from  emotional  aspects 
Confidence,  courage  to  make  a  decision 
Provide  vision  of  outcome  /  set  priorities 
Leadership 
Communication 
Teamwork 
Compassion 

Capability  to  influence  and  collaborate 
Experience  working  with  the  response  team  or  network 

Experience  with  strategic  decisions 

Understand  and  navigate  the  political  nature  of  the 
incident 

Have  relevant  technical  domain  knowledge 


Cognitive 

Cost  /  benefit  analysis 

Skills 

Interpersonal 

Skills 


Experience 


Domain 

Knowledge 


Fig.  2.  Skills  emphasized  by  SMEs 


•  The  skill  is  easily  taught  or  measured  for  currently; 
or  that  it  is  not  a  critical  skill  for  a  specific  type  of 
emergency  manager  (i.e.  it  is  not  of  high  importance 
to  FEMA  IMAT  emergency  managers  but  it  may  be 
critical  to  a  first  responder). 

•  The  skill  may  still  need  to  be  prioritized  through 
conventional  preparation  techniques  and  included  in 
evaluations. 


We  interpret  a  low  mention  rate  to  mean  that  these  skills 
do  not  need  to  be  a  focus  of  augmentation  of  training  and 
evaluation,  and  thus  they  are  of  lesser  interest  to  this  project’s 
goals. 

The  skills  emphasized  by  the  SMEs  have  been  organized 


into  four  categories: 

•  Cognitive  Skills  are  mental  abilities  that  aid  in  per¬ 
forming  data  processing  tasks 

•  Interpersonal  Skills  refers  to  skills  a  person  uses  to 
interact  with  others 

•  Experience  is  having  prior  opportunities  to  practice  or 
directly  encounter  a  situation 

•  Domain  Knowledge  refers  to  information  retention 
relevant  to  a  given  subject 

Figure  3  shows  the  relative  proportion  of  reports  of  skills 
according  to  these  four  categories.  There  were  149  total  skill 
reports  (the  sum  of  the  counts  in  Figure  2).  The  majority 
of  skill  reports  were  for  skills  classified  as  cognitive  (89 
reports),  about  a  quarter  of  reports  were  for  skills  classified  as 
interpersonal  (35  reports),  and  the  remainder  were  classified  as 
either  experience  based  (14  reports)  or  domain  knowledge  (11 
reports).  This  breakdown  suggests  that  extensions  to  current 
training  and  preparation  methods  should  primarily  focus  on 
targeting  cognitive  and  interpersonal  skills.  Note  that  this  does 
not  mean  that  knowledge  and  experience  based  skills  are  not 
important  (indeed,  there  is  no  doubt  that  they  are  essential), 
merely  that  they  are  already  well  addressed  by  current  training 
and  preparation  and  thus  were  not  emphasized  as  an  area  of 
concern.  Also  note  that  both  of  those  categories  contained 
skills  that  individually  rated  high  on  the  list,  indicating  value 
in  better  addressing  all  four  areas  in  training. 


Domain 

Knowledge 

8% 


Fig.  3.  Breakdown  of  skill  reports  by  type  of  skill 
C.  Observed  Trends 

While  most  of  the  skills  identified  were  valued  and  de¬ 
scribed  consistently  by  all  types  of  interviewees,  some  of  the 
responses  strongly  correlated  to  the  role  that  the  professional 
holds.  We  observed  trends  between  rad/nuc  SMEs  and  emer¬ 
gency  managers  in  operational  decision  making  roles,  trends 
between  National  vs.  Regional  FEMA,  and  trends  between 
federal  roles  overall  and  state/local  roles  overall. 

1 )  Rad/Nuc  SMEs  vs.  Operational  Decision  Makers:  We 
interviewed  a  range  of  rad/nuc  SMEs  with  extensive  knowl¬ 
edge  on  INDs  but  without  direct  experience  in  operational  de¬ 
cision  making  roles,  including  IND  planners,  health  physicists, 
and  public  health  professionals.  Their  assessment  of  decisions 
and  skills  were  closely  coupled  to  those  in  operational  roles  in 
most  regards,  suggesting  that  the  theoretical  and  practical  sides 


of  the  community  are  in  alignment.  In  one  aspect,  however,  the 
two  groups  routinely  differed. 

Almost  every  decision  on  the  list  requires  some  under¬ 
standing  of  INDs  or  radiation,  indicating  that  all  SMEs  placed 
importance  on  technical  knowledge.  However,  not  all  SMEs 
called  it  out  as  an  area  of  concern  in  its  own  right,  and 
the  relative  emphasis  of  technical  knowledge  versus  other 
skills  differed.  The  rad/nuc  SMEs  tended  to  identify  a  lack  of 
basic  technical  knowledge  about  radiation  among  their  highest 
concerns  with  current  preparedness.  Several  explicitly  said  that 
they  trusted  that  the  people  filling  emergency  management 
roles  had  all  the  soft  skills  and  proficiencies  necessary,  but  they 
believed  those  emergency  managers  lacked  basic  familiarity 
with  radiation  or  with  the  network  of  SMEs  available  to 
provide  such  information.  In  contrast,  emergency  managers  in 
decision  making  roles  (of  all  levels)  were  much  more  likely 
to  emphasize  the  relative  importance  of  the  cognitive  and 
interpersonal  skills  needed  to  effectively  apply  the  technical 
knowledge,  and  more  likely  to  downplay  the  presence  of  the 
technical  knowledge  as  a  critical  gap  (or,  at  least,  downplay  it 
as  a  gap  that  is  hard  to  fill  with  traditional  training  methods). 

Several  of  the  rad/nuc  SMEs  noted  that  the  necessary 
technical  material  can  be  covered  in  a  classroom,  but  they 
worried  that  such  methods  did  not  result  in  the  knowledge 
being  retained  since  the  majority  of  responders  did  not  have 
much  opportunity  to  practice  or  apply  IND-related  knowledge. 
Their  concerns  suggest  a  slightly  different  opportunity  that 
augmented  training  techniques  might  support  improving  re¬ 
tention  of  technical  knowledge  by  providing  more  concrete  and 
interactive  rehearsal  without  a  full  scale  exercise. 

2)  National  vs.  Regional  FEMA:  National  level  FEMA 
professionals  consistently  emphasized  navigating  politics  care¬ 
fully,  building  cohesive  and  effective  teams,  and  being  aware 
of  the  advisory  role  that  FEMA  plays  to  state  and  locals. 
Regional  FEMA  and  federal  domain  SMEs  consistently  em¬ 
phasized  preparation  and  training,  trusting  SMEs  over  political 
advisors,  and  the  importance  of  making  good  decisions.  Table 
II  provides  a  comparison  between  typical  responses  from  the 
regional  and  national  level  FEMA  responders. 

Given  the  lengthy  experience  of  all  the  interviewees,  we 
assume  that  the  discrepancies  shown  in  Table  II  are  not  an 
indication  of  one  group  being  right  or  more  knowledgeable 
and  that  the  two  stances  are  not  contradictory  but  instead  differ 
in  their  emphasis.  We  interpret  the  differences  in  response 
as  reflecting  real  differences  in  the  skills  and  priorities  most 
pertinent  to  different  types  of  emergency  managers.  From 
this  characterization,  we  can  start  to  form  a  picture  of  how 
training  for  such  skills  might  be  different  for  these  distinct 
roles,  and  how  alternative  methods  may  be  required  to  meet 
the  needs  of  these  groups.  Recognizing  these  differences  could 
be  especially  valuable  when  supporting  professionals  moving 
between  national  and  regional  roles. 

3)  Federal  vs.  State/Local:  There  are  strong  similarities  in 
how  the  federal  and  state/local  participants  prioritized  critical 
skills.  Many  of  the  skills  (e.g.  timely  decisions,  resource  allo¬ 
cation,  and  selective  attention)  ranked  highly  by  both  groups 
were  described  in  nearly  identical  manners.  For  other  skills, 
the  same  underlying  competency  was  described  as  manifesting 
in  different  ways,  reflective  of  the  varying  roles  local,  state. 


TABLE  II. 


Comparison  of  regional  and  national  level  FEMA  responses 


Subject 

Typical  Regional  FEMA  Stance 

Typical  National  FEMA  Stance 

Decision  Making 

It  is  important  to  make  the  right  decision 
quickly,  as  the  early  stages  of  the  incident 
are  critical. 

It  is  important  to  maintain  an  advisory  role 

until  asked  to  do  otherwise. 

Basis  for  Strategic 
Decisions 

Strategic  decisions  should  be  informed  by 
factual  knowledge  and  aimed  to  save 
lives.  They  should  not  be  swayed  by 
political  factors,  which  are  distractions. 

Strategic  decisions  need  to  be  reflected  in 
organizational  structures  and  take  into 
account  the  social  and  political 
environment  for  them  to  be  effective. 

Setting  Priorities 

Be  ready  to  make  hard  decisions  to 
maximize  the  ultimate  goal  of  saving 
lives. 

Priorities  differ  in  different  situations,  and 
it  is  important  to  understand  and 
communicate  your  goals. 

Key  Skills 

Decision  making  under  pressure,  without 
sufficient  information,  and  with  no 
perfect  options. 

Organizing  teams,  bringing  SMEs  to  bear 
on  the  appropriate  problem,  and  creating  a 
cohesive  response. 

and  federal  emergency  managers  play. 

For  example,  for  the  skill  ’’Ability  to  adapt  plans  and 
improvise  solutions  based  on  changing  conditions”  (ranked 
3rd  among  cognitive  skills  in  the  interviews),  the  federal- 
level  emergency  managers  generally  emphasized  the  impor¬ 
tance  of  adjusting  a  prepared  plan  as  the  incident  unfolds 
and  improvising  on  the  fly.  Conversely,  the  state  and  local 
emergency  managers  generally  emphasized  adapting  a  plan 
to  the  particular  needs  of  an  agency  or  community  prior 
to  an  incident.  We  tallied  both  responses  as  a  single  skill, 
defined  as  the  ability  to  interpret  a  plan’s  intent  and  decide 
when  to  execute  it  as  written  and  when  to  modify  it  based 
on  incident  specifics.  We  believe  the  differences  are  worth 
noting,  as  they  could  affect  the  design  of  training  materials 
or  exercises  targeting  the  same  skill  for  different  types  of 
emergency  managers. 

A  similar  pattern  was  observed  with  the  skills  ’’Determine 
reliable  information  and  sources”  (ranked  2nd  among  cognitive 
skills  in  the  interviews)  and  ’’Experience  working  with  the 
response  team  or  network”  (ranked  1st  among  experiential 
skills).  Both  skills  describe  the  importance  of  having  and 
knowing  how  to  request  the  network  of  SMEs  that  can  be 
leveraged  for  technical  guidance  for  decision  makers.  Federal 
emergency  managers  generally  described  these  skills  in  terms 
of  identifying  SMEs  and  integrating  teams  under  the  pres¬ 
sure  of  an  incident.  State/local  emergency  managers  generally 
described  this  integration  in  terms  of  building  social  and 
organizational  connections  prior  to  an  incident. 

Considering  the  roles  that  federal  vs.  state/local  emergency 
managers  typically  play  in  an  incident  and  the  types  of 
incidents  they  typically  encounter,  these  differences  are  not 
surprising.  Federal  responders  work  in  support  of  state/local 
teams  when  the  incident  is  abnormal  or  of  a  scale  that 
overwhelms  the  available  local  resources.  Federal  teams  may 
be  called  on  to  support  a  wide  range  of  incidents,  so  plans 
and  networks  must  be  revised  and  learned  on  the  fly,  as  it 
is  not  possible  to  rehearse  with  every  possible  partner  and 
scenario.  In  contrast,  the  state/local  teams  deal  with  more 
routine  incidents,  allowing  them  the  opportunity  to  collaborate 
with  the  same  agencies.  In  such  cases,  it  is  feasible  to  adapt 
plans  and  build  networks  ahead  of  time  to  preempt  needs  that 
arise  during  the  incident.  If  anything,  the  surprising  result  is 


not  that  the  two  groups  differ  in  their  expression  of  those 
skills,  but  that  their  different  needs  boil  down  to  such  similar 
underlying  skills.  Hence,  similar  materials  can  likely  be  used 
to  augment  training  for  both  types  of  emergency  managers,  but 
those  materials  will  need  to  be  grounded  in  different  examples 
and  exercise  scenarios. 

IV.  Conclusions  and  Next  Steps 

This  document  has  described  an  analysis  of  the  key  deci¬ 
sion  points  and  corresponding  skills  where  current  training  and 
preparation  may  leave  emergency  managers  and  responders 
unprepared  during  an  IND  incident  or  other  large  scale  disaster. 
Through  reviewing  documentation  and  conducting  interviews 
with  26  SMEs  and  survey  responses  from  11  SMEs,  we  can 
better  understand  common  threads  in  the  difficult  decisions 
in  an  IND  incident  response  and  the  corresponding  skills 
possessed  by  experienced  responders  who  make  them.  Unique 
challenges  and  decisions  exist  in  an  IND  incident  response,  but 
the  underlying  skills  identified  as  addressing  those  decisions 
were  not  IND  specific  and  would  likely  apply  to  any  moderate 
to  large  scale  incident.  We  observed  that  most  critical  decisions 
are  made  especially  difficult  by  time  pressure,  amplifying  the 
importance  of  cognitive  and  interpersonal  skills.  If  time  were 
not  a  constraint,  then  all  of  the  requisite  information  could  be 
gathered,  the  team  could  come  to  a  consensus  on  each  issue 
as  it  arose,  and  the  task  would  be  a  more  straight-forward 
technical  question. 

When  we  look  at  the  decision  making  skills  emphasized  by 
the  SMEs,  cognitive  and  interpersonal  skills  were  highlighted 
significantly  more  than  technical  knowledge  and  experience. 
As  stated  in  the  Skill  Analysis  section,  we  believe  that  this 
is  because  all  responders  who  are  in  important  emergency 
response  positions  already  have  the  necessary  technical  knowl¬ 
edge  and  experience.  These  skills  are  effectively  taught  and 
selected  for  using  the  traditional  methods  of  classroom  and 
on-the-job  training,  although  there  are  concerns  that  retention 
may  be  poor  for  skills  that  are  not  applied  in  practice  (such 
as  IND-related  knowledge).  The  cognitive  and  interpersonal 
skills  emphasized  by  expert  emergency  responders  are  much 
more  difficult  to  quantify  and  teach.  Those  skills  are  currently 
developed  only  over  years  of  experience,  limiting  the  pool  of 
qualified  emergency  managers  and  making  it  difficult  to  fill  all 
the  decision  making  positions.  We  see  this  gap  as  a  possibility 
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to  bring  training  and  evaluation  of  cognitive  and  interpersonal 
skills  through  the  use  of  serious  gaming,  and  to  improving 
the  retention  of  relevant  technical  knowledge.  Several  SMEs 
expressed  the  sentiment  that  well  run  exercises  can  address 
those  needs,  but  that  exercises  are  a  time  consuming  way  of 
accumulating  knowledge  and  require  SMEs  to  be  taken  away 
from  their  posts  to  participate.  We  also  hope  to  explore  the 
ability  for  techniques  and  technologies  from  gaming  to  provide 
a  less  burdensome  format  for  exercises  or  as  augmentation  to 
existing  exercise  regiments. 

A  central  concept  of  the  larger  effort  is  identifying  how 
to  use  repeatable  and  interactive  experiences  outside  of  the 
field  of  radiological  and  nuclear  disaster  management  to  help 
disaster  management  professionals  develop  and  demonstrate 
relevant  expertise  [1],  The  next  steps  in  the  project  will  be 
to  investigate  how  the  findings  from  this  initial  report  can 
inform  the  development  of  training,  evaluation,  and  research 
platforms.  These  platforms  may  be  in  the  form  of  strategic 
games  to  help  individuals  and  teams  of  radiological  response 
officials  and  emergency  managers  rapidly  improve  pertinent 
behavioral  competences  or  rehearsal  exercises  augmented  with 
gaming  techniques  to  impose  reduced  burden  on  participants. 
The  effort  will  explore  classifying  what  types  of  game  me¬ 
chanics  are  most  suited  to  different  types  of  skills  pertinent  to 
emergency  management. 
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